test(otel): cover replay and chained execution - #796

Add OpenTelemetry conformance handlers for user-function and callback contexts (22–23), invocation retries (24), caught callback-failure replay (25), external callback completion across later replays (26), and a chained child that suspends and resumes (27). Both views exercise the public Core 3 / OTel 2 APIs.

Merge prerequisite: #868 remains required for the ordinary local Case 26 tests. Keep these failures as merge gates; do not skip, xfail, or suppress them. The Case 25 fixes (#865 and #870) are already included from main. Cases 25–26 remain active in both cloud suites and the local suite. Case 21 retains its local replay tests but is explicitly NOT_IMPLEMENTED in both cloud views.

Validated review comments

Design

• No public SDK API, runtime behavior, or new configuration is introduced. The review changes remove four handler source lines and obsolete workflow controls.
• Probes use ordinary OpenTelemetry spans without supplying or repairing their parents. Existing trace/parent/count assertions detect absent or incorrect SDK context; redundant preflight rejection only adds a durable-callback failure path.
• Test callbacks record observations instead of raising assertions that the SDK can retry. Case 26 asserts exactly [("target", 1)] before checking history gates. Later-barrier and final raw-count assertions remain. Case 22 keeps synchronized sibling probes and checks both callback and invocation-host context restoration after execution.
• Existing OTel branch tests use different handlers. Four test-only barriers remain necessary to force overlap in the actual Case 22 child, parallel, and map callbacks. They neither inject nor repair tracing context.
• Case 23 retains its unrelated-helper-failure and unexpected-success checks. Both protect required scenario execution and have negative public-runner coverage.
• Case 21 remains deferred because the shared history requirements order InvocationCompleted before WaitSucceeded, while the timer can finish first. The recorded cloud run observed the reverse order on a successful execution. Increasing the wait only reduces the race. Re-enable cloud coverage after the shared requirement accepts both orders while preserving replay assertions.

Workflow and cloud evidence

The reusable workflow is pinned and maintained by Dependabot. The test catalog defaults to the conformance repository's main, with conformance_test_ref retained for manual overrides. Both PR and manual runs execute invocation and execution suites. All 13 remaining caller inputs and 7 secrets match the pinned schema; its default case count is 27. Legacy long-running templates and local contract tests remain, but there are no launch/check controls in this caller.

The previous four-job cloud run reported 25 passed, Case 25 failed on ERROR span status, and Case 21 NOT_IMPLEMENTED in every job. Case 26 already passed in the cloud. The saved detailed Case 25 histories verify the empty-error shape against AWS Lambda. Metadata-only Error: {Truncated: true} is a local-runner expectation and remains unverified against the service. The updated revision's cloud qualification is pending; a green run still would not claim Case 21 coverage.

The additional Case 17 callback-error regression remains outside this PR. Its source is preserved in 6484e577; correct its detailed/metadata-only history expectations before restoring it. The existing Case 17 handler remains active. The Case 11 fix is already merged in #864.

Validation

On Python 3.13.5 and Hatch 1.17.0:

• Baseline at 10b1b31: 91 passed, 2 failed. Both failures are Case 26, reported as shifted history IDs after an assertion triggered an SDK step retry.
• Updated branch: 92 passed, 2 failed. Both now directly report target completion must reach the exporter before the observed step, with [("target", 0)] instead of [("target", 1)].
• With fix: record callback terminal operation updates #868 production changes temporarily applied: 94 passed. Those changes were then restored and are not included in this PR; fix: record callback terminal operation updates #868 remains a separate prerequisite.
• Command: hatch run test:all packages/aws-durable-execution-sdk-python-conformance-tests-otel/tests .github/scripts/tests/test_opentelemetry_conformance_workflow.py -qq --tb=short.
• Workflow regressions: 2 failed / 5 passed against the old caller; all 7 pass after pinning and removing obsolete controls.
• The same 10 public-runner context/parent tests pass before and after guard removal. Mutations that remove SDK parenting still cause 4 Case 22 failures and 2 Case 23 failures. A separate context-restoration mutation causes 4 direct pytest failures. All mutations were restored.
• Conformance-package formatting/lint, workflow-test formatting/lint, commit-message lint, and whitespace checks pass. Independent review found no actionable issues.

Existing test expectation changes

• Template/catalog contract tests (test_templates_map_every_otel_requirement, test_templates_declare_only_the_expected_functions, test_function_names_identify_the_requirement, test_makefile_builds_every_function, test_every_registered_handler_exists, test_chained_invoke_functions_point_at_their_target, test_handler_modules_are_exactly_the_expected_set, test_active_callback_and_retry_handlers_keep_their_mapping, test_deferred_requirements_are_explicit) include active cloud cases 22–27, Case 27 targets, and retained local-only Case 21. Only Case 21 is pending.
• test_callback_probes_keep_their_actual_sdk_parent adds unexpected helper-failure/success cases while keeping successful-scenario parent/count assertions.
• test_external_callback_completion_precedes_later_replays checks (value, export_count) outside the step; the observation point and required count remain unchanged.
• test_user_function_probes_keep_sdk_context_across_resume checks recorded callback/host context identities outside SDK retry handling and requires the expected observation counts.
• test_opentelemetry_conformance_caller_uses_current_workflow_contract expects a full commit SHA; a new manual-input test prevents reintroducing inactive controls.

Case 27 companion PRs: JS #1068 and Java #824.